👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

cc @K-Mistele

The function partial_json_parser will complete the partrial JSON string {"name": "tool_name", "arguments": {"arg1": " to {"name": "tool_name", "arguments": {"arg1": ""}}, which will result in an error when calculating the incremental changes in the arguments field.

Thanks everyone for making this potential fix!
I'll try on my own examples as soon as tomorrow
In the spirit of testing on the same basis, is it possible to have a minimal a very minimal python codebase requesting vllm with tools and streaming ?

Thanks everyone for making this potential fix! I'll try on my own examples as soon as tomorrow In the spirit of testing on the same basis, is it possible to have a minimal a very minimal python codebase requesting vllm with tools and streaming ?

For the relevant code, please refer to the following link: #10395

Thanks everyone for making this potential fix! I'll try on my own examples as soon as tomorrow In the spirit of testing on the same basis, is it possible to have a minimal a very minimal python codebase requesting vllm with tools and streaming ?

For the relevant code, please refer to the following link: #10395

Totally missed that, thanks a lot!

Please fix the linter errors.

Don't worry about DCO, we can pass it manually if you agree to it.

Please fix the linter errors.

I've fixed the linter errors.

I used the hermes_tool_parser.py as tool-parser-plugin and registered the parser as hermes_patched, but still have the same problem.

Traceback (most recent call last):
  File "/app/hermes_tool_parser.py", line 228, in extract_tool_calls_streaming
    function_name: Union[str, None] = current_tool_call.get("name")
                                      ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'
Error trying to handle streaming tool call.
Traceback (most recent call last):
  File "/app/hermes_tool_parser.py", line 292, in extract_tool_calls_streaming
    args_delta_start_loc = cur_arguments_json.index(delta_text) \
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: substring not found

Here is how I start vllm service:

python3 -m vllm.entrypoints.openai.api_server \
--model /app/Qwen2.5-72B-Instruct-AWQ \
--port 7415 \
--tensor-parallel-size 2 \
--gpu-memory-utilization 0.95 \
--max-model-len 64000 \
--enforce-eager \
--disable_custom_all_reduce \
--enable-auto-tool-choice \
--tool-parser-plugin /app/hermes_tool_parser.py \
--tool-call-parser hermes_patched  \
--chat-template /app/qwen.jinja

I used the hermes_tool_parser.py as tool-parser-plugin and registered the parser as hermes_patched, but still have the same problem.

Traceback (most recent call last):
  File "/app/hermes_tool_parser.py", line 228, in extract_tool_calls_streaming
    function_name: Union[str, None] = current_tool_call.get("name")
                                      ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'
Error trying to handle streaming tool call.
Traceback (most recent call last):
  File "/app/hermes_tool_parser.py", line 292, in extract_tool_calls_streaming
    args_delta_start_loc = cur_arguments_json.index(delta_text) \
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: substring not found

Here is how I start vllm service:

python3 -m vllm.entrypoints.openai.api_server \
--model /app/Qwen2.5-72B-Instruct-AWQ \
--port 7415 \
--tensor-parallel-size 2 \
--gpu-memory-utilization 0.95 \
--max-model-len 64000 \
--enforce-eager \
--disable_custom_all_reduce \
--enable-auto-tool-choice \
--tool-parser-plugin /app/hermes_tool_parser.py \
--tool-call-parser hermes_patched  \
--chat-template /app/qwen.jinja

I use vllm0.6.3.post1 docker compose, and entrypoint is:

entrypoint: ["/bin/sh", "-c", "python3 -u -m vllm.entrypoints.openai.api_server --model /data/models/Qwen2.5-72B-Instruct-AWQ --enable-auto-tool-choice --tool-call-parser hermes --tensor-parallel-size 2 --gpu_memory_utilization 0.97 --max_model_len 20000 --max_num_seq 40"])